Sync translog to remote on primary activate #10839

sachinpkale · 2023-10-23T04:44:15Z

Description

Currently, we don't guarantee translog sync to remote on primary mode activation.
This creates issue in flows similar to one listed below:
- Index some data with 0 replica
- wait for snapshot to be created
- delete index
- restore from snapshot - wait for index to turn green
- terminate instance
- _remote/restore the index — Fails.. shards are not getting assigned. — recovery is failing with exception: java.nio.file.NoSuchFileException: <index path>/4/translog/translog.ckp
In this change, we make sure to always sync translog on primary activation.

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-10-23T05:17:39Z

Compatibility status:

Checks if related components are compatible with change 27245ff

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-10-23T05:21:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28758/
CommitID: e38e455
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/index/shard/IndexShard.java

Signed-off-by: Sachin Kale <[email protected]>

github-actions · 2023-10-23T07:17:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28777/
CommitID: 2f57b5b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Sachin Kale <[email protected]>

server/src/main/java/org/opensearch/index/shard/IndexShard.java

Signed-off-by: Sachin Kale <[email protected]>

github-actions · 2023-10-23T08:20:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28784/
CommitID: 6a64edb
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-23T08:52:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28788/
CommitID: e0a596a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Sachin Kale <[email protected]>

github-actions · 2023-10-23T10:48:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/28806/
CommitID: 7af013e

codecov · 2023-10-23T10:50:18Z

Codecov Report

Merging #10839 (27245ff) into main (51626d0) will decrease coverage by 0.06%.
Report is 14 commits behind head on main.
The diff coverage is 90.15%.

@@             Coverage Diff              @@
##               main   #10839      +/-   ##
============================================
- Coverage     71.31%   71.25%   -0.06%     
+ Complexity    58671    58667       -4     
============================================
  Files          4860     4869       +9     
  Lines        276335   276451     +116     
  Branches      40198    40198              
============================================
- Hits         197068   196988      -80     
- Misses        62803    63015     +212     
+ Partials      16464    16448      -16

Files	Coverage Δ
...upport/replication/TransportReplicationAction.java	`77.10% <100.00%> (-3.58%)`	⬇️
...ava/org/opensearch/cluster/node/DiscoveryNode.java	`91.62% <100.00%> (+0.17%)`	⬆️
...a/org/opensearch/common/network/NetworkModule.java	`92.20% <100.00%> (+0.20%)`	⬆️
...rg/opensearch/common/settings/ClusterSettings.java	`92.85% <ø> (ø)`
.../java/org/opensearch/gateway/GatewayMetaState.java	`69.25% <100.00%> (+0.73%)`	⬆️
...earch/index/remote/RemoteStorePressureService.java	`93.22% <ø> (-6.78%)`	⬇️
server/src/main/java/org/opensearch/node/Node.java	`85.31% <100.00%> (+0.09%)`	⬆️
...ting/admissioncontrol/AdmissionControlService.java	`100.00% <100.00%> (ø)`
...issioncontrol/controllers/AdmissionController.java	`100.00% <100.00%> (ø)`
...g/admissioncontrol/enums/AdmissionControlMode.java	`100.00% <100.00%> (ø)`
... and 14 more

... and 451 files with indirect coverage changes

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteRestoreSnapshotIT.java

ashking94

LGTM

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteRestoreSnapshotIT.java

Signed-off-by: Sachin Kale <[email protected]>

github-actions · 2023-10-23T13:07:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled

URL: https://build.ci.opensearch.org/job/gradle-check/28816/
CommitID: 27245ff
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-10-24T09:04:32Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-10839-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6f36752d9e84e95ce2280347cc26b0c9138b2d57
# Push it to GitHub
git push --set-upstream origin backport/backport-10839-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-10839-to-2.x.

--------- Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>

--------- Signed-off-by: Sachin Kale <[email protected]>

--------- Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

sachinpkale added the skip-changelog label Oct 23, 2023

Bukhtawar reviewed Oct 23, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/shard/IndexShard.java Show resolved Hide resolved

Sync translog to remote on primary activate

2f57b5b

Signed-off-by: Sachin Kale <[email protected]>

sachinpkale force-pushed the translog-sync-on-primary-activate branch from e38e455 to 2f57b5b Compare October 23, 2023 06:46

sachinpkale marked this pull request as ready for review October 23, 2023 06:46

Fix failing tests

6a64edb

Signed-off-by: Sachin Kale <[email protected]>

ashking94 requested changes Oct 23, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/shard/IndexShard.java Show resolved Hide resolved

server/src/main/java/org/opensearch/index/shard/IndexShard.java Outdated Show resolved Hide resolved

Do not throw exception on failure

e0a596a

Signed-off-by: Sachin Kale <[email protected]>

Fix failing test

7af013e

Signed-off-by: Sachin Kale <[email protected]>

ashking94 reviewed Oct 23, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteRestoreSnapshotIT.java Outdated Show resolved Hide resolved

ashking94 approved these changes Oct 23, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteRestoreSnapshotIT.java Show resolved Hide resolved

Address PR comments

27245ff

Signed-off-by: Sachin Kale <[email protected]>

This was referenced Oct 23, 2023

Re-factor post processing of activate primary mode into a listener #10863

Open

Add support to create empty local translog if remote translog is empty #10854

Closed

sachinpkale requested a review from Bukhtawar October 24, 2023 03:36

Bukhtawar approved these changes Oct 24, 2023

View reviewed changes

sachinpkale merged commit 6f36752 into opensearch-project:main Oct 24, 2023
18 checks passed

sachinpkale added the backport 2.x Backport to 2.x branch label Oct 24, 2023

opensearch-trigger-bot bot added the backport-failed label Oct 24, 2023

sachinpkale added a commit to sachinpkale/OpenSearch that referenced this pull request Oct 24, 2023

Sync translog to remote on primary activate (opensearch-project#10839)

bb38f7f

--------- Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>

sachinpkale mentioned this pull request Oct 24, 2023

Sync translog to remote on primary activate (#10839) #10882

Merged

sachinpkale added a commit to sachinpkale/OpenSearch that referenced this pull request Oct 25, 2023

Sync translog to remote on primary activate (opensearch-project#10839)

08b059f

--------- Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>

gbbafna pushed a commit that referenced this pull request Oct 25, 2023

Sync translog to remote on primary activate (#10839) (#10882)

1d92801

--------- Signed-off-by: Sachin Kale <[email protected]>

ashking94 mentioned this pull request Nov 25, 2023

[Remote Store] Handoff refreshes, translog uploads during relocation from old to new primary #11330

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sync translog to remote on primary activate #10839

Sync translog to remote on primary activate #10839

sachinpkale commented Oct 23, 2023 •

edited

Loading

github-actions bot commented Oct 23, 2023 •

edited

Loading

github-actions bot commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

codecov bot commented Oct 23, 2023 •

edited

Loading

ashking94 left a comment

github-actions bot commented Oct 23, 2023

opensearch-trigger-bot bot commented Oct 24, 2023

Sync translog to remote on primary activate #10839

Sync translog to remote on primary activate #10839

Conversation

sachinpkale commented Oct 23, 2023 • edited Loading

Description

Check List

github-actions bot commented Oct 23, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Oct 23, 2023 • edited Loading

Codecov Report

ashking94 left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Oct 24, 2023

sachinpkale commented Oct 23, 2023 •

edited

Loading

github-actions bot commented Oct 23, 2023 •

edited

Loading

codecov bot commented Oct 23, 2023 •

edited

Loading